<html>
<head><meta charset="utf-8"><title>UI tests: redundant error checking? · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html">UI tests: redundant error checking?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="224915097"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224915097" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224915097">(Feb 02 2021 at 18:31)</a>:</h4>
<p>Hey, I'm trying to update some UI tests with new output after a PR of mine changed it -- and I'm running into some usability trouble. In particular I'm trying to update <a href="https://github.com/rust-lang/rust/blob/b81f5811f96fe750ab28c15219d1b0dba6b1dc90/src/test/ui/symbol-names/issue-75326.rs">https://github.com/rust-lang/rust/blob/b81f5811f96fe750ab28c15219d1b0dba6b1dc90/src/test/ui/symbol-names/issue-75326.rs</a>, because some of the meaningless hash values in symbol names have changed. Handling such values seems to be solved for <code>.stderr</code> files by normalizing them via a regex, e.g.:</p>
<div class="codehilite"><pre><span></span><code>//[legacy]normalize-stderr-32bit: &quot;h[\d\w]+&quot; -&gt; &quot;SYMBOL_HASH&quot;
</code></pre></div>
<p>This is a good solution. However, there are also error message annotations in the source file itself, like:</p>
<div class="codehilite"><pre><span></span><code>//[v0]~^^^^  ERROR symbol-name(_RNvXINICs4fqI2P2rA04_11issue_75326s_0pppEINtB5_3FooppENtB5_9Iterator24nextB5_)
</code></pre></div>
<p>They do <em>not</em> seem to be normalized -- which means I have to hardcode symbol hashes again.</p>
<p>My question is: why are these annotations there in the first place? All of their output seems to also be checked in the <code>.stderr</code> file. Is this just a historical leftover?</p>
<p>If yes, that would be great. I can just remove them and make the <code>.stderr</code> files via normalization rules.</p>



<a name="224915680"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224915680" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> tm <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224915680">(Feb 02 2021 at 18:35)</a>:</h4>
<p>Not really answer to the why question, but the annotation itself can use an arbitrary substring of the message.</p>



<a name="224915692"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224915692" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224915692">(Feb 02 2021 at 18:35)</a>:</h4>
<p><code>//~ ERROR string</code> will work even if <code>string</code> is a subset of the error message, so volatile things like hashes can be omitted.</p>



<a name="224915907"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224915907" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224915907">(Feb 02 2021 at 18:37)</a>:</h4>
<div class="codehilite"><pre><span></span><code>//[v0]~^^^^  ERROR symbol-name(_RNvXINICs4fqI2P2rA04_11issue_75326s_0pppEINtB5_3FooppENtB5_9Iterator24nextB5_)
</code></pre></div>
<p>If I'm not mistaken, this doesn't contain a hash to be normalized, this is just the v0 mangling of the symbol (which is typically what's being tested in this part of the test suite, right?).</p>



<a name="224915914"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224915914" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224915914">(Feb 02 2021 at 18:37)</a>:</h4>
<p>Yes, I just discovered that removing the annotations in the source files make <code>compiletest</code> complain that there are unexpected errors. I'm not sure if that makes sense since comparing the <code>.stderr</code> files will also catch unexpected errors.</p>



<a name="224916055"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224916055" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224916055">(Feb 02 2021 at 18:38)</a>:</h4>
<p><span class="user-mention" data-user-id="116107">@davidtwco</span> The <code>Cs4fqI2P2rA04_</code> part contains a "disambiguator" that has changed. It is as opaque as a hash.</p>



<a name="224916087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224916087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224916087">(Feb 02 2021 at 18:38)</a>:</h4>
<p><span class="user-mention silent" data-user-id="124287">mw</span> <a href="#narrow/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F/near/224916055">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="116107">davidtwco</span> The <code>Cs4fqI2P2rA04_</code> part contains a "disambiguator" that has changed. It is as opaque as a hash.</p>
</blockquote>
<p>Fair enough</p>



<a name="224916154"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224916154" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224916154">(Feb 02 2021 at 18:39)</a>:</h4>
<p>It <em>is</em> more stable than legacy <del>crate</del> symbol hashes, but it's still an implementation detail</p>



<a name="224916282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224916282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224916282">(Feb 02 2021 at 18:40)</a>:</h4>
<blockquote>
<p>My question is: why are these annotations there in the first place? All of their output seems to also be checked in the .stderr file. Is this just a historical leftover?</p>
</blockquote>
<p>It's not a leftover, it's a redundancy to avoid mistakes.<br>
It's hard to match lines in <code>.stderr</code> files with lines in the source code + since <code>.stderr</code> files are generated automatically it's very easy to generate them and forget to review whether that wall of text matches the intent or not, but manual review is necessary for reliable testing.<br>
<code>//~ ERROR</code> annotations ensure that expected lines produce errors and unexpected lines do not produce errors.</p>



<a name="224916361"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224916361" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224916361">(Feb 02 2021 at 18:40)</a>:</h4>
<p>But, yes, I'll follow <span class="user-mention" data-user-id="352985">@tm</span> and <span class="user-mention" data-user-id="123856">@Vadim Petrochenkov</span> suggestion and make the annotations more generic. Thanks!</p>



<a name="224916442"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224916442" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224916442">(Feb 02 2021 at 18:41)</a>:</h4>
<p><span class="user-mention" data-user-id="123856">@Vadim Petrochenkov</span> Yes, I can see that it would be easy to accidentally <code>--bless</code> something that's actually wrong</p>



<a name="224916958"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/UI%20tests%3A%20redundant%20error%20checking%3F/near/224916958" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/UI.20tests.3A.20redundant.20error.20checking.3F.html#224916958">(Feb 02 2021 at 18:45)</a>:</h4>
<p>I just remembered that we allow the string <code>[...]</code> to act as a wildcard in debuginfo tests. Something simple like that would already help quite a bit.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>